Research + Planning * Professional Development 
for California Community Colleges LEADERSHIP. EMPOWERMENT. VOICE. 


La theRPgroup af seer lad ee 


AB 705 Research and Analysis 
Ideas for Collaboration between 
Researchers and Faculty 


This document provides ideas for collaboration between English, Math, and credit ESL 
faculty and institutional research, planning, and effectiveness (IRPE) professionals in the 
California Community Colleges to examine local impacts of AB 705. The ideas from this 
document were generated and collated from regional workshops and events hosted by 
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stakeholders, as well as workshops and events hosted by the RP Group in spring and fall 
2019. The document begins by sharing opportunities for collaboration identified by 
English, Math, and credit ESL faculty and IRPE professionals, followed by commonly- 
used definitions and terms. The document concludes with a list of considerations for 
both English, Math, and credit ESL faculty and IRPE professionals as they work together 
in the context of AB 705 evaluation. 


Opportunities for Collaboration 


e Include both faculty and IRPE staff in departmental and college-wide AB 705 
implementation and evaluation structures 


e Collaboratively review the role of MIS codes in collecting data and review the re-coding 
developed to track AB 705 changes with faculty identifying the courses and IRPE 
professionals working to code them appropriately 


o Jointly identify all English, reading, credit ESL, mathematics, quantitative reasoning, 
and other appropriately-related discipline courses that satisfy transfer, graduation, 
and/or local degree requirements and ensure they are correctly coded! and 
reported to the state Chancellor’s Office 


o Jointly identify coding for support versus target courses in order to determine 
efficacy of support formats for specific groups and in order to validate whether 


1 At time of publication the MIS DED was located here: https://www.cccco.edu/About-Us/Chancellors- 
Office/Divisions/Digital-Innovation-and-Infrastructure/Management-Information-Systems/Data-Element-Dictionary 
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support courses should be required, optional, embedded or some other 
modification, as well as to determine the trade-offs among varying types of support 


e Faculty invite IRPE colleagues to attend department/division/discipline meetings so they 
can better understand curricular structures, sequences, and changes in order to help fully 
evaluate outcomes and comparison coursework 


e Jointly develop a research plan and agenda in collaboration with your AB 705 
workgroup/steering committee and/or with faculty from each department/discipline 
(English, Math, credit ESL, and any other with courses meeting requirements in these 
disciplines) to ensure learning is taking place, student outcomes are improving, and equity 
gaps are closing. 


o Title 5 requires statistical validation for any required prerequisites or corequisites. A 
research plan to evaluate programmatic and placement changes in response to AB 
705 should be in place prior to implementation, or as early as possible. Discipline 
and department practitioners should be nimble and responsive to the outcomes, 
adjusting their newly-implemented plans as needed to optimize success for all 
students. 


o IRPE practitioners should be open to faculty’s ideas for data collection and show 
faculty the data currently available to be included in an analysis, and help them 
consider the implications of data that are or are not available. 


o Faculty and IRPE practitioners should consider the idea of an extensive research plan 
that can be accomplished over the long-term, including both short- (e.g., course 
success, throughput rates, next course success, enrollment trends, financial aid, 
academic standing) and long-term outcomes (e.g., degree, transfer and 
employment) of students in English, math, and credit ESL. 


o Both quantitative (e.g., success rates) and qualitative (e.g., student and faculty 
experiences) data should be used to understand the impact of changes that have 
taken place. 


© IRPE professionals and English, Math, and credit ESL faculty should collaboratively 
agree upon a comprehensive research agenda with defined cohorts prior to the 
analysis to help reduce bias or the perception of bias. 


o Jointly consider that changes in placement practices were intended to improve 
student outcomes and opportunities while closing equity gaps. Consider unintended 
consequences and address them as soon as possible. Examine the impact of changes 
on disproportionately impacted groups and monitor equity gaps. 


e Discipline faculty and IRPE practitioners should work together to develop surveys and/or 
focus groups to gather feedback from students and faculty including: reasons for drops or 
withdraws, perceived appropriate placement, level of support provided, type of support 
provided, etc. When deciding about the use of surveys and/or focus groups, consider the 
following: 
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© Faculty may choose to be surveyed on their perceptions around student ability, 
discipline-specific supports, and additional needs such as professional development, 
etc. 


o Part-time faculty may have a different set of needs, concerns, or feedback than full- 
time faculty 


© Focus groups may be a good way to take a deeper dive into those areas where a 
survey may not provide that opportunity 


o Include a variety of disciplines outside of English, math, and credit ESL in order to 
understand the scope across the institution 


e Invite IRPE practitioners to present regular updates at AB 705 meetings and/or English, 
Math, and credit ESL department/division meetings 


e IRPE professionals should share early drafts of findings and reports with discipline faculty 
leaders to gather feedback and make applicable adjustments. 


Common Definitions and Terms for Outcomes 
Monitoring 

e Course success = At, A, A- B+, B, B-, C+, C, P grades 

e Course non-success = D, F, NP, |, W grades 

e Course success rate = successful grades / enrollment 

e Course withdrawal rate = students with W grade / enrollment 

e Drop rate = students who dropped with no grade / total enrollment including drops 


e Throughput rate = Number of students who start at a particular level of English or math and 
successfully complete the degree- or transfer-level course appropriate to their educational 
goal or program of study in two semesters or three quarters (math and English) or who 
successfully complete either a transfer-level English (TLE) course or an ESL course 
equivalent to transfer-level English (TLEE) in six semesters or nine quarters (credit ESL). 


e Volume of successful completions = Total number of students who successfully complete 
the degree- or transfer-level course appropriate to their educational goal or program of 
study compared to the total number of students who successfully completed the course 
prior to the change 


e Volume of successful completions (ESL) = Total number of credit ESL students who 
successfully complete either a transfer-level English (TLE) course or an ESL course 
equivalent to transfer-level English (TLEE) compared to the total number of students who 
successfully completed prior to the change in six semesters or nine quarters 
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Degree- or transfer-level course enrollment rate (English and math only) = Percent of 
students who enroll in basic skills courses in a particular term or year / percent of students 
who enroll in the degree- or transfer-level course 


Volume of enrollments in degree or transfer level course (English and math only) = Total 
number of students who enroll in the degree- or transfer-level course compared to the total 
number of students who enrolled in the degree- or transfer-level course prior to the change 


Students with a degree or transfer goal: students whose educational goal is degree or 
transfer based on their application or an updated intake process or other campus-specific 
process 


Additional ESL-Specific Definitions 


ESL: English as a second language 

ESOL: English to speakers of other languages 

ESL student: a student enrolled in an ESL course 

ELL: English language learner 

ELD: English language development (used in high schools to mean ESL coursework) 
TLE: Transfer-level English course 

TLC: Transfer-level English Composition 

TLEE: ESL equivalent course to transfer-level English composition (aka, ESL TLC) 


International student: a student who attends an American college with an F1 or FZ visa 


Data Considerations 


Potential Comparison Groups 


Fall 2019 compared to fall 2018, or 2017-18 academic year compared to 2018-19 


Throughput rates for all first-time students who completed the degree- or transfer-level 
course within one academic year compared to first-time students who completed the 
course (or a higher level course) in the fall by starting level (e.g. fall 2017 cohort tracked for 
one year compared to the fall 2019 cohort tracked for one term) 


o Be sure to include throughput rates based on where the student started in the 
sequence: number of students who started one level below (CB 21 = A), two levels 
below (CB 21 = B), three levels below (CB 21 = C) 


Disaggregate outcomes by special population groups to determine disproportionate impacts 
(e.g., DSPS students compared to non-DSPS students in the same course in the same term) 
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Success of students in the degree or transfer course by high school GPA band (default 
placement rules) compared to success rates of students in the same GPA band in the 
corequisite course 


Calculating ESL Throughput Rates 


Throughput rates for all first-time students who completed transfer-level English (TLE) or 
ESL equivalent to transfer-level English (TLEE) within a three-year timeframe compared to 
first-time students who completed TLE/TLEE by the credit ESL level in which they first 
started (one, two, three, four, five or more levels below transfer level) for fall 2016 cohort 
tracked for three years compared to the fall 2015 cohort tracked for three years, etc. 


o Be sure to include throughput rates based on where the student started in the 
sequence: number of students who started one level below (CB 21 = A), two levels 
below (CB 21 = B), three levels below (CB 21 = C), four levels below (CB 21 = D), five 
levels below (CB 21 = E), and six levels below (CB 21 = F). 


o If you have a noncredit sequence, you may want to include the throughput rates for 
students transitioning from noncredit to credit ESL. Note that the three-year 
timeframe for AB 705 starts when degree/transfer seeking students take their first 
credit ESL course. 


Student Variables 


Need to disaggregate outcomes, when sample sizes are appropriate, by different student groups to 
determine disproportionate impacts, including but not limited to: 


Racial/ethnic groups 

Economically disadvantage groups 

Gender 

Part-time / full-time 

DSPS, EOPS, CalWorks, Veterans, Foster Youth status, etc. 


Course attempts (i.e., first time enrolled in the course, repeating the course the first time, 
second time, etc.) 


Simultaneous enrollment in regards to rigor and units of other classes 
Enrolled or placed into ESL course prior to enrollment 

By high school GPA band 

By high school attended 


High school courses taken and grades earned 
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How long to enrollment in English or math (i.e., did they enroll in math or English in their 
first term enrolled, second term, etc.) 


Low-income or first-generation status 

PELL or Promise Grant status 

STEM or non-STEM major or course taking patterns 

Any other disproportionately-impacted groups as identified in the local equity plan 


Success after first or second repeat — are there differences between completion after the 
first and second attempt in basic skills courses historically vs. direct placement into degree 
or transfer level 


o Differences in completion if repeat is with support, without, or one-level below, etc. 
Placement level 
Highest education level 


Course attempts (e.g., first time enrolled in the course, repeating the course the first time, 
second time) 


Educational goal (e.g., degree, transfer, basic skills development) 


Students starting in basic skills math or English or noncredit ESL and transitioning to credit 
vs. those starting in credit 


Additional ESL-Specific Student Variables 


Country of origin 

Number of years of English instruction in high school or prior 
Number of years of high school completed 

English language development (ELD) courses taken in high school 
Graduated from a U.S. high school or foreign high school 

TOEFL score (for international students) 

Number of years residing in the U.S. 

International student with F1/FZ visa vs non-U.S. citizens 
Comfortable speaking English (variable in CCCApply) 

Student moving from English to ESL 


ESL students taking only credit ESL courses 
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ESL students enrolled in a math course or general education courses (GE) simultaneously 


Course Characteristics 


Modality (face-to-face, hybrid, online) 

Accelerated, integrated or compressed curriculum 

Time of day and/or day of the week 

Embedded supports such as tutors or counselors 

Course units (particularly with high-unit corequisite course) 

Term (fall, winter, spring, summer) 

Term length (number of weeks) 

Corequisite recommended vs. required 

Stretch or late start course 

Departmental standards or common final vs. each instructor sets own standards and exams 


Type of corequisite course (i.e., lecture vs. lab, taught by the target course instructor or 
another instructor, contextualized or embedded across disciplines, cohort, integrated) 


Students self-select or recommended into corequisite courses 
Early alert systems 


Type of credit ESL course: Reading, Writing, Speaking, Listening 


Instructor Effects 


Ensure faculty anonymity in the results by aggregating terms until 10 or more sections are available for 


analysis 


Success rates by section 
Distribution of letter grades by section 
Equity gaps by section 


Type of innovation or strategies used in the classroom (i.e., flipped class, just-in-time 
remediation, lab or lecture, etc.) 


Student success in next course in the sequence or other General Education (GE) courses 
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Long-term Outcomes 
Compare a cohort of students prior to AB 705 implementation to a cohort post-implementation. 


e Success in next course in the sequence (i.e., Precalculus 1 to Precalculus 2 or Calculus, 
ENGL1A to ENGL 1B or 2) 


e Throughput to transfer-level English (TLE) or ESL course equivalent to transfer-level English 
(TLEE) 


e Complete a degree or certificate 

e Transfer to a four-year institution 

e Did not pass after multiple attempts 

e Completion of degree- or transfer-level English AND math within first year of enrollment 
e Loss of financial aid eligibility or placed on academic probation 


e Enrollment in and completion of other GE courses and timing of enrollment (e.g. were 
students able to take other GE courses in their first year at a higher rate than in the past) 


e Impacts of additional corequisite units (i.e., heavier unit loads) 
e Success within major-related courses 
e Wage information and/or employment outcomes 


Questions or comments? Please contact ASCCC at info@asccc.org or RP Group at 
research@rpgroup.org 
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